[SkipRecovery] Re-enable mixed-delete tests on Iceberg 1.10.2+ [reduced-it] - #15964
[SkipRecovery] Re-enable mixed-delete tests on Iceberg 1.10.2+ [reduced-it]#15964wjxiz1992 wants to merge 2 commits into
Conversation
Signed-off-by: Allen Xu <allxu@nvidia.com>
There was a problem hiding this comment.
🟢 Approval recommended
No unresolved blocking issues were identified.
Pull request overview
Re-enables mixed Iceberg delete validation for Iceberg 1.11+ while preserving quarantine behavior for older or unknown versions.
Changes:
- Gates xfail and fixed-seed markers by Iceberg version.
- Disables AQE for the v3 count-plan assertion.
- Resets Spark configuration before cleanup.
File summaries
| File | Description |
|---|---|
integration_tests/src/main/python/iceberg/iceberg_merge_on_read_test.py |
Updates mixed-delete test gating and session cleanup. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
 |
|
build |
|
|
||
| pytestmark = iceberg_unsupported_mark | ||
|
|
||
| # Iceberg 1.11.0 fixes cached equality-delete records being interpreted in the wrong field order: |
There was a problem hiding this comment.
Can the quarantine gate account for Iceberg 1.10.2 as a fixed runtime? Apache backported the equality-delete schema-ordering fix to the 1.10.x branch and released it in 1.10.2.
Signed-off-by: Allen Xu <allxu@nvidia.com>
wjxiz1992
changed the title
|
build |
|
build |
JaCoCo production line coverage: not fully measurable locally — no compatible Scala 2.13 / shim 411 nightly baseline is published.
Contributes to #12885.
Description
This test-only change restores enforced CPU/GPU result checks for mixed Iceberg position and equality deletes on Iceberg 1.10.2 and newer. Both format-v2 and format-v3 tests currently allow failures and permanently pin their data seed, even when running with the upstream equality-delete cache fix. Production behavior is unchanged. The historical mismatch remains quarantined on older or unknown Iceberg versions; no count or row mismatch was observed on the tested 1.11.0 runtime.
Apache Iceberg #15514, released in 1.11.0 and backported to 1.10.2, fixes equality-delete records being cached in a query-dependent field order. This PR:
SHOW TABLESdoes not inherit the scan-only assertion.Older supported dependencies are not fixed by this PR, so #12885 should remain open.
Validation
Review follow-up: 44 actual decorator/seed-policy combinations passed, including the 1.10.1/1.10.2 boundary, later releases, and an unknown runtime. Re-ran all six mixed-delete cases plus runtime/shim detection on Spark 4.1.1 / Iceberg 1.11.0 with seed
1749483297: 7 passed (v3 includes OOM injection). Iceberg 1.10.2 was policy-checked only; it was not GPU-runtime-tested locally.Local Spark 4.1.1 / Scala 2.13 / Iceberg 1.11.0, Hadoop catalog, Python 3.10.18, UTC, RTX 5880 Ada:
1749239638).1749483297and forced OOM injection: 7 passed. Each reader returned CPU=GPU counts of 626 (v2) and 549 (v3), with full-row equality also passing.20260911: 7 passed.mvn packagefordist,integration_tests: BUILD SUCCESS. Validation used production base31db6361829e8cbd258be94eda2842b667aef048; the later PCBS-only merge does not overlap this patch.Reduced premerge collection retains all six recovered cases: each changed test has only the reader-type parametrization. Databricks is unaffected because the existing test harness explicitly skips Iceberg tests there. No production performance change.
JaCoCo execution data was captured during the forced-OOM run. Measured module deltas: none;
sql-plugin,iceberg,delta-lake,shuffle-plugin, andudf-compilerdeltas are N/A without a matching baseline. Published shim-350/401/404 reports cannot establish a shim-411 delta; no partial total is claimed.AI assistance: The change and PR description were prepared with Codex assistance.
Checklists
Documentation
Testing
(Please provide the names of the existing tests in the PR description.)
Performance